The transition model is used to imagine possible outcomes of taking the action/option, which are then evaluated by the value functions to change the policies and the value functions themselves. This process is called planning. Planning, like everything else in the architecture, is expected to be continual and temporally uniform. On every step there will be some amount of planning, perhaps a series of small planning steps, but planning would typically not be complete in a single time step and thus would be slow compared to the speed of agent-environment interaction. Planning is an ongoing process that operates asynchronously, in the background, whenever it can be done without interfering with the first three components, all of which must operate on every time step and are said to run in the foreground.

For anyone familiar with Reinforcement Learning, it is hard not to know who Richard Sutton is. The Sutton & Barto textbook is considered canonical in the field. I always find it highly inspirational to study the views of genuine thought leaders. Thus, when they present a new research vision, I'm primed to listen. This summer, Sutton and his colleagues Bowling and Pilarski outlined a research vision for Artificial Intelligence, designing a blueprint for their research commitments in the next 5 to 10 years. The full document is only 13 pages long and comprehensively written, so it doesn't hurt to have a look.

Many aspects of geosciences pose novel problems for intelligent systems research. Geoscience data is challenging because it tends to be uncertain, intermittent, sparse, multiresolution, and multi-scale. Geosciences processes and objects often have amorphous spatiotemporal boundaries. The lack of ground truth makes model evaluation, testing, and comparison difficult. Overcoming these challenges requires breakthroughs that would significantly transform intelligent systems, while greatly benefitting the geosciences in turn.